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ABSTRACT 

This report describes the instructional guality 
inventory (IQI) , vhich was developed to provide guality 
control/eva-luation procedures for the instructional systei i 
deyelopsent (ISD) • The ISD is described as a Method used by the 
military to develop or revise training .courses. IQI procedures are 
detailed in six sections. Section 1 explains the IQI classification 
systen, which dLassifies instructional parts (objectives, tests, and 
presentation's) according to performance tasks and instructional 
content. Section 2 reviews the IQI steps, vhich are used to determine 
if each objective is adequate for further instructional development. 
The third section outlines the process used to determine whether t^st 
items are consistent with objectives and whether each item is 
adequate. The following section provides guidelines for insuring that 
the presentations are consi'Sten^ with, the objectives and the test 
^ items and are adequate, in section 5 rules for. assessing presentation 
adequacy are presented. The 'final section reviews the resources 
nipeded to use the IQI. (Author/CSS) . " 
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FOREWORH 

The Instructional Quality Inventory (IQI) was. developed in support of 
Navy Decision Coordinating Paper, Education and Training Development 
(NDCP-Z0108-PN),/ under subproject P.30A, Adaptive Experimental Approach to ' 
Instructional Design, and the sponsorship of the Director of Naval Education 
and Training (OP-99). The overall objective of the subprojpct 1s to develon 
an empiricably-based instructional design support/ system to aid developers 
in deciding on instructional alternatives based on cost/benefits and specified 
resource 1 ipiitations. the purpose of the IQI,'wh.ich vas briqinally called 
the Instructional- Strategy Diagnostic* Profile (ISnP), is to .provide o'jality 
control and/or evaluation procedures for instructional developdient. 

* 

A number of reports have been published on the IQI/ISnp. The first pro;- 
vided ah interim traininn manual for the ISOP (ffPRnr Special Renort ?7-14), 
dnd the next,three addressed its pmpirical and workshop evaluations (PPRHC 
technical Reports 77-25 and 77-43 and Special Report 78-17). As a result 
of these ovaluations, the' ISnP was extensively revi^stfd and retitled as 
thP! IVil. These revisions were ijicluded in NPRDC Technical Note 78-5, 
which- provided anMnterim training manual for Jhe lQI. 

This report is the first in a series of four on IQI procedures. The remain- 
irtflthFee reports, to be^ublished in early 1979, include: 



^^1. AnMQl User's Manual , wh'ich will include a complete description of all 
IQirprocedures, and provide examples of their use. ^ 

2. An IQI Training Workbook, which will provide additional examples, alonq 
with practice on the ICH procedures. • ^ 

- ■ » ^ 
. 3. An IQI Job Performance Aid, which will provide a brief version of each 
IQI 'procedure. 

When th^s'? four reports have>^eGn published, previous training manuals (MPRnr. 
Special Report 77-14 and Tec hVipal Nate, 78-5) will have been superseded. 

TheilQI is intended for use by the .Chief of Naval Education and Traininq; 
the Chie^ of Naval Technical Training; the Chief of Naval Education anH Trainina 
S.upport (ispecif ically, the Instructional Program Development Centers); the 
Conniande^ Traininq Command, Atlantic; the Commander Training Command, Pacific; 
and all other Navy activities concerned with the deveVopment, revision, or 
acquisitidn of instructional programs. Prospective users of the IQI are invited 
to contact. this command for assistance in implementation. 



^DONALD F. PARKED 
Comma nd i ng jOff i cgr 



SUMMARY- 

Probl em - 

Modern military instruction is developed according to a systematic 
method called Instructional Systems Development (ISD). This method in- 
cludes the. following steps: 

1. Job/task analysis' leading to specification of traln-ing'objectives. 

2. Development of tests to measure student progress toward the ob/ectives 

. • ■ A 

3. Design of new instriJction and/or adaptation of, existing instfirction 
to achieve the objectives. ' ^ ' 

4. Implementation of the training program. 

5. Evaluation and feedback for course maintenance. 

Various military activities are using i:his method to develop or revise 
many of their training courses. Thus, it. is im(x)rtant that qual ity \pontrol 
and/or evaluation procedures be developed for use with ISD. Such procedures 
are needed so that: ' - 

■• ■ • ■ ■ ' ' ■ , ^ • 

J«. Quality- can be maintained throughout instructional design, thus, errors 
made early in-^fhe development process will noj be magnified as development pro- 
ceeds. ' . ' . . ... 

2. Existing materials can be evaluated with respect to newly derived tra,in»- 
ing objectives, so that they can be^modified or revised if necessary. 

3. Performanee def icierrcies of coorse graduates can be traced to poss,ible 
deficiencies in instructional materials. • ^ - 

4. -instructional materials obtained through -contract effortscan be evaluated 
before tney are accepted. , 7 . ► • " 

Puriatose ' - 

^ ' . 

THe purpose Of this research and development effort was to develop quality 
control/evaluation procedures for the- three main products of an instructional 
development effort; that is, objectives, tests, and instructional materials or 
presentations, these procedures are intended for use by military instructional 
design and development personnel. • ' ^ " , 

Background ' " * ' ^ ' . 

The Instructional Strategies Diagnostic Profile (ISDP) was originally developed 
by Courseware; Inc. under contract to' the Navy Personnel Research and Development 
Center (Mer^rill, Richards, Schmidt, & Wood, 1977hMts. purpose was to provide a 
method for diagnosing defects in instructional matekial^J ' 



0 ^ 

vii 



Finally, the IQI is intended for use by people who are familiar with ISD; 
it cannot be used by untrained personnel. Also, its application depends upon . 
a good task analysis' or the availability of subject matter experts--and pref- 
erably both. This is because the IQlXassumes that what needs to be taught 
has already been determined. , 

This report provides an introduction to and overview of the IQI procedures. 
It is designed to acquaint managers of instructional development efforts, in- v 
struction evaluators, and contract monitors with the IQI. While it provides 
a substantive overview of the IQI process, it is not acomplete IQI training 
program. 

The fol lowing. reports in the IQT series will.be available in early 1979: 

1. A User's Manual, which will include a complete description' of all IQI 
procedures, and provide examples of their use. 

2. A Training Workbook, which will provide additional examples and practice 
on the IQI procedures. 

3. A Job. Performance Aid, which'^will provide a brief version^^of each IQI 
procedure. 

^ Conclusions . ' 

In previous empirical tests and in field workshops, 'it has been shown that 
the IQI is an effective method for assur*infl the quality of instructional pro- 
grams. In addition, there is great user irrt^r^est in the IQI; over 300 copies 
of previous versions of the IQI have- been distributed, and workshops on earlier 
versions have been given 'for many Navy commands. ^Based on feedback from work- 
shop participants, the IQI appears to be\a very useful addition to Navy instruc- 
tional design and develof^jji^t methods. 

Recommendations . \ 

1. This report, and the forthcoming IQI user and training* manual s, should 
be published as NAVEDTRA publ ication§, .and made an official part of Navy ihstruc- 
tional desi>gn and development procedures. ' ^ 

2. The IQI shbuld be considei^d for use by Training Program Coordinators 
in the NayJil Technical Trainict^ Comand'^ (and other conriands concerned with train- 
ing management) as an evaluation am quality contrdl tool. 

3. The IQI procedures should be. considered for adoption in the Instructional 
Program Development Centers under the Chief\of ^Naval Education and Training 
Support. ; ^ # > , 

4. The IQI pro^dures should.be considered for use by technical monitors 
of contracts fqr trair^ing programs as evaluation/acceptance tools. 

J 5. The IQI procedures should b^ considered for^ inclusion in instructor 
jtrairriag for those instructors who Will have curriculum development responsibility. 
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INSTRUCTIONAL QUALITY INVENTORY PROCEDURES 



FICAtlcL 



THE CLASSIFICATION SYSTEM: 

The following olaaBifiontion pyatem ie ueed in all IQI prooeduma. 
, It to applied to the three main parto of inatruotion: objeotivea, 
teato, and instructional pveapntatione. 

t i ^ 

Each objective, test item, or pifece .of presentation, can be classified 
according to: 

1. What the student must do, i.e., the TASK tb be performed, and 
^ ■ - •\ 

2. The type of information thfr student must learn, i.e., the 
instructional CONTENT.." 



V CONTENT 




TASK 



In the IQI, these two-classification dimensions have been combined 
to form the TASK/CONTENT MATRJX . 
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THE TASK dimension: 

There are two main TASKS a , student can perfor^:, ^- 

1. He can REMEMBER Irvfomiatlon, or 

2. He can USE the Information to do something 



REMEMBER 



USE 



{ 



EXAMPLE : 

Here are\wo test items, 

l/^ The symbol for resistor 1*s 



( 2. vUSjing your knowledge of electronic theory, what would happen 

In the circuit shown below 1T the load resistance were 
shorted? 

These two test items differ with respect ^to what the student is 
• supposed to do (TASK). In number %he student has to REMEMBER 
something, and in number 2, the student has to apply or USE his 
knowledge in a new situation. 



% 



CONTENT dimension: 



Ther^ are fUe types of COf^TENT: 



I' 



REMEMBER 
USE 



FACT CONCEPT PROCEDU^ RULE PRINCIPLE. 

If 



are simple associations between jiames , objects, symboV!^, 
rocati(^s , etc. \ 

CONCEPTS are categories or classifications defin^ by certain 
specified characteristics. 

PROCEDURES consist of ordered sequences of steps or operations 
performed on a single object or in a^pecific situation, v . 

RULES also consist of ordered sequences of operations, but can 

be performed on a variety of objects or in a variety of situations. 

Pf^NCIPiES involve explanations or predictions of why things 
happen in the world. That is, they concern predictions or 
interpretations based on theoretical or cause-effect relatidpships. 



NOTE: Facts can only be remembered, 
remembered or used. 



The others cccn be 
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Th0 /f>//()wtrz£/ 
REMEMBER tapk 



REMEMBER FACT 



REMEMBER CONCEPT 



REMEMBER PROCEDURE 



REMEMBER RULE 



REMEMBER PRINCIPLE 



level: 

\ 

/• Thcoymbpl for mxinfov ii\ 
P.. Tho\fltudont Dill Hot th- namiU^Vif (Jw 
payrtn in tiw uimi iruiti^tt twi fniU rictn-nt . 

// biaf ^h^d^fimncj c/wracf of d jt*f pwrtj}, 
P.. Thi^ :'tu(mmt Dili define th* tkirinuif ktndo of 

I. Liot in oviicr the (\tc})i\ Jhr cleaning jxn hf-le rtflr 
.P,\ Thi* ittudcnt un U deanrinc t hr prorn.^/iVrf^* for 
preparincj and eending a xYidio mf'iWiUja, 

1. Liet the ateps involved in finding the rhumbs 
line couroe between Ujo poirif^ on the mrth. 

2. The vtudent un U atate the j^fies^il rule for ^ 
oolving for nirnutt ourref.r , voltage 
and reeietance, 

L State the pi^inciplee of elr,'t\>6ri mcwement in a j 

aemi (conductor junction., 
P. The student i^U veoall the rruaorio whxj Hydraulic- 

fluid oontamtnation nn4(^t bo ^ \Hded. 



Facts can only be remembered, hut for thp ^ihi^r content fypes, the 
student may be asked to USE his h^owledgr i^hiQuify, perfom, solve ^ 
or predict. The follow^ng ar^ exami.^les of the USE task level for ^ 
all content types except facts: • ^ . , 



USE CONCEPT 

USE Vroceoure 

USE Wt- 



USE PRINCIPLE 



2, 



I. 
2. 



^ L 



Which of the pumps aboa^r! .%ip*are Jet pumps? 
Given photographs of clouds, \ the student will 
sort them aacovding to typ^e ^cumulus, stratus, etc.'), 

Clean an- M-16 rifle. 

The ^tudent uitl prepare and send a radio message. 

Calculate the rhwrp^line course from Peart 
Harbor to Long Beck^hi j 
Given the values for voltage and resistance , ^he * 
student Dill calculate the current flow. ^ ^ 

Describe the thec^tical movement of electrons in 
a PNP ^tran8istor.'\ • ' 

The student will predict whfit is likely to occur y 
if the landing gea\ Jfluid were contaminated. 

■ ■ 
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THt l)S( IIVK CAN BE rURrHLK Ul VlDflll INTO TWO !YI]LS: 

1. IJSl -lINAIDfl) in which the student hds no aids cxcopt f'*^ own niorT«ry. - 

2. USf -AIDIl) In which the student has' a Job did for performing the task. 

/'.>r USE-AIDf D CONCtPTS thf .lid ^i^hmld .•.>n;u\t^ yy uV.t'/i/.'M 

lU r\i( t'iji^ , ifu^ludi'n^j riii^h i*rt(t\*nl -/m*/ fh$' dt^rfiUf^*} 

hi in'-rrTjdr ^\uuui, t hr did tniij tml it itu^Indr a ! i i*hn\u*t t'Vi i\t ; 

For USE-AIDED PROCEDURES thr ,n.{ i.\>uJ,i h,- ,i /int ,-r ^. 

Fi^v USE-AIUt^D RULES fhr iiid i^^util h' at IrmM / \\t ,i t rrth^nt nj* f^ir 
fnr^mii: (^r vuli' to /»r 'ipplird, urui t^otifil irududc ijuidt'firuu^ Jot* 
tjhrn and hc^u^ (o ^jpjdij ft, v 

rnr USE-AIDED PRINCIPLES the aichLkmh! <iLno at huiat a iMAifcmry^t 

thp pHyh*rrh\ and <^ould inc^Iudi' ^ju^dr I zmui f\)r uh-n and h^tJ 
^^ iipfdl^ . . 



USE-AIDED: 



USr^AfOED: 



1 

A [dlot 'it pv^^flight ^^h^ckt ii}t tn a USE-A^D 
f^'t\H\\iuve , Thr f>tlot do*jr> not ^tiw to rerm^iht^r ,0u' 
iifcpii t)v their order htu^ausu^ ^hrij are jm^Tlir (*hi'c*k- > 
Ht^t, Thr pilot doci} need to prrfoj^r/ thr nteps 
<*orrt'f*t I ij , r 

''Thr ctwicnt will firld-ntrip an M-in riflr." 
firra, thr atudent muct mmcrfibcr thr r^teps in thr 
<*nrrrct ordcr^ anil prrfc^rm them i^ovvri*t lij. 



In sunmary. the gEMEMBER level involves ''pure" remembering, 

the USE-UNAIDED level involves remembering what is to be used, 
and then using it, and 

/ the USE-AIDED level, involves "pure" using. 
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THE Eniffi TASK /^CPNT£NTJ1ATRIX- Jg^§^^^^^^ b^lowT 



I 



i 



FACT 

RECALL OR 
RECCTGtllZE 
' NAMES^ 
PARTS^ 
DATES^ 

PLACES'^ 5TC. 



PROCEDURE ^ 

SEmJEI«:EvOF 
STE^S REMEM- 



^CONCEPT-^^. 

•REMEMBER 
CHARACTER I S- 
TICS> OR 
CLASS IJFY OB- 
JECTS^ EVENTS 
DR IDEAS AC- 
CORDING TO 

CHARACTER IStlCB EQUIPMENT 



RULE^ * PRINCIPLE 

REMEMBHR Oft -.REMEMBER^ OR 
USE A ^SEQUENCE INTERPRET 
BERED OR USED OF STEPC* WHICH PREDICT^ WHY 

A SINGLE APPLY. ACXRjpSS OR HOW THINGS 
SITUATION" OR 'S/TUATIONS HAPPEN^^OR*" 
ON. A SINGLE ^ OR ACROSS CAUSE-EFFECT 
PIECE OF EQUIPMENTS ' ^jRiTLATIONSHIPS 



REKEflBER - R€call or • 

RECOGNIZE FACTS> CON-' 
' CEPT DEFINITIONS, STEPS 
- OF PROCEDURES OR RULES^ 

STATEMEfTTS QF PRINCIPLE 






• 






USJE- UNAIDED - tasks which require 

r CLASSIFY ING^ PERFORMING A PROCEDURE/ 
USING A RULE^ EXPLAINING OR PREDItlTING 
WITH NO AIDS EXCEPT MEMORY. 

USE^^^ED --SAME AS USE-UNAIDED^ 
EXCEPT JOB AIDS ARE AVAILABLE. 

: vi 














^\ 

\ 





Apy objective, test item, or ftf^'ece of instruction will be 
^classifiable intone and only one cell of the matrix^ above. 

This matrix is" used in all IQI steps. 
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OBJCCTlVE ADEQUACY; • . • ' . . , ' ■ 

^ , The first step pn, the IQI ^proce^ure co^respondB'with i^e development ' 
J * ^of training objectives. The procedure described betcHi) is used to 
>K^^^determin^^f each objective is -adequate for further instruo^tional 
development^. ^ - , , • . , • 

Objectives areT ADEQUATE if they satisfy three general criteria: 

1. . Is' the 'object iv.e CORRECTLY STATED? Does the objective include statements^ 

of actions the student is to perform after training y ^ the cohdi^tions under 
. which thia ^performance is expeajbedy .and the Ssi^ndavds which the performance 
must meet? If even^one of these parts, is.ptissingy the objeative^s 
' * inadequate because tr&ining for it cannot be designed or evaluated. 

EXAMPLE, : Inadequate objective: ^^The student will prepare' a standard 

Navy messageV^ - This is inadequate because it does not specify *' 
^either the conditions (given a typewriter? TTY?) or the 
; . standards (how fast and how many errors). 

2. Is the 'objective CLASSIFIABLE on the task/content matrix?.^ 7/ the' ob- 
jective cannot be classified^ this means that the actioyr the student 
is to perform is not stated clearly enough so that we know what the 

' student is to do. Training cannot be designed or evaluated in these 
, circtonstances . 

I EXAMPLE : The objective ^'The student will learn repair procedures fqr 
I . the XYZ' radar set^* is not classifiable. It is not clear 

whether the student should remember; the procedures or actually 

use them. 

3. Is the "intent" of the objectivfc APPROPRIATE for the purpose of the ' 
course? The actions conditions^ and standards specified in the. * 
objective should b'e as close as possible to the actions ^ conditions^ 
and standards of the task to be performed on the job after training. ' 

In addition y it' ^s as&umed that the ultimate jUntent^^ of any 
training program is to teach the student how to dd\pme thing (i.e. ^ US^E 
level). Therefore^ there must be a USE-level objective for each REMEMBER 
objective. (Facts are a spe<^^l case: Although facts are not lised^ they 
ofter^ must be taught to provtde tx knowledge ba^e for a later use^level 
task. UTiereforey in order to, justify teaching facts^. they must support 
some use-level objective. ) , . 

Conversely^ USE-UN4;iDED tasks should be taught at the REMEMBER ^ 
level before_ being taught at the USE level. Therefore^ ju^ as every 
REMEMBER objectiife should have a corresponding' USE objective^ every USE 
Qbjeqtive should have a previous REMEMBER objective. 

• EXAMPLE :. The objective ^^The student will identify the connection of a 

vbltmeter to measure the voltage across a component by selecting 
an illustration^^ is not appropriate foi* the intent of the course. 
The student will not me possible illustrations of connections 
, - on the joby but will be required to set up t)^ connection^ thus- 

the action, should he revised. 



TEST CONSISTENCY and ADEQUACY: / . 

Once ohiectives are adequate y test items can he developed. The. next 

\ IQI 8^ep is the qiAality^ control step for t^st developmi/ii*- This step 

involves determining whethe^ test items are C0I^SISJENT/ii;it/2 objectiveei^ , 

/ oncT whether ehch item is ADEQUATE, ♦ ' V ^ . * 

A test item is CONSISTENT with its objective if: ' 

!• The ACTION (TASK/CONTENT level) of the test item is thg same as^that 
of the objective. 

-I r . / ^ 

2; The CONDITIONS under which the item is administered are as close. a§ 
possible to those of the objective. 

3. The STANDARDS-in the^ test item, or the STANDARDS for scoring th^s* item, 
are as close as possible to the standards in the objective. * .J 

// . r . , 

EXAMPLE : Objective: Given the necessary tools^and an operator's 

manual^^he student w%ll set up and operate 
a double-acting, reciprocating pumpy in five 
minutes and according to the manual specifications. 

Inconsistent test item: "List tjie steps of procedure for 
^tarting, operating, and stopping a double- . 
acting reciprocating pump. '' . ) 

This test item is^ inconsistent , because 
its TASK/CONTENT is RBMEMBER-PROCEDURE ' 
instead of USE-AIDED-PROCEDURE.^ Notice 
. that the action the student is* to perform 
in thS test is. not the same as the action . 
/ required in' the objective. 

Consistent test it0n: ''Use the operator's manual y and 

/ necessary tools to set up and operate a double- 
" ; acting reciprocating pump. , You will pass this 

■N * test if you complete this task within 5 minutes, 

I . • in accordance with the manual specifications." 

/ * This test item is consistent with the 

J" ' ^ ' • ■ ' objective. Notice , however , that if • 

either the conditions or grading standards 
had been left outy the item would have 
been inconsisten\ ^ 



. / 
/ 

/ 

/ 



9 4. The FORMAT of the test item is appropriate for the TASK/CONTEMT 
level. 

• " ^EXAMPLE: "The steps imthe procedure for operating a 

Qet pionp are listed/helcw. Arrca^igf tpem in 
- ■ * the correct omierJ^ 

* * This is an iriajppropriate formaf for 
' , - \ REMEMBER-PROCEMJRE because the siMdent 

V ' doesn't have ta remember the prooedt^ey onty * " ^ 

^ recognize the steps. ^ , 

. . ■ r » ^ f x.*-^ 

.Note: Recognition items (multiple ohoide^ matching^ pn^e-false) are 
usually HOI appropriate test formats for REUBMBER level objectives. ^ 
This is because these items do not reflect typical job-performance 
' requirements. / . , . ■ ' ^ ' 

' . Multiple choice y matching ^ and true- false items are appropridte j 
for concept recognition^ and can be appropriate for USE level objectives 
if they, are ca-A fully designed. However ^ for USE level objectives, 
"hands-^on^' performance, tests are usually most appropriate.^' > 

After a test item is consistent with its objective, the test item is . 
ADEQUATE if: ' " ^ 

1. • The item is clear and unambiguous.' 

• ■ » ■ 

2. The item does not give away its own answer ornhe answer to' any 
other item on the test. ^ 

3. The item is well constructed. !^ 

4^ Other adequacy concerns covered in the IQI manual are met. 

EXAMPLE : ' "Which of the follouyLng ..." is ambiguous 

because it does not say "choose all that apply" 



or "choose the best 



n 

< • • 
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PRESENTATION CONSISTENCY: ■ 

At this poi^nt in the IQI process ^ dbjeotives are adequate^ test 'items 
are danaietent with objectives^ and te&t items are adequate. The 
next instructional design step is tojprepare tl^e instructional materials 
' or presentations. ^ The next IQI step is to insure that the preserCtations 
are CONSISTENT with the objectives and test it^msj and are ADEQUATE. 

In the previous sectidn^ determining test^objective consistency involved 
aimiparing eacns^B&i^tem with its related objective. Determining 
PRESENTATION CONSISTttJCY, hn the other hand, involves checking whether 
or not each of the IVSTHUCTIONAL COMPONENTS required for a given , 
objective-- test item Hs. present. There are different types of instruc- 
tional ^components. ,In order to insure consistency j the appropriate - 
components must be present for each TASK/CONTENT level. Not all } • 
task/ content levels require all compiments. . . ^ 

The Instructional PRESENTATION cWoN^^ are: . 

1, STATEMENT : ' The instruction tells the student something he must^ learrii 

2* EXAMPLES ; The instruction shows the USE of content (concept j procedure j 
• rule or principle). I I 

3. ^ PRACTICE : Th^ student practices REMEMBERING or USING the content j and 
is given feedback. \ 



PRESBNTATION COMPONENTS:^ ^ " ^ ■ ' 

STATEMENT Component; The instruction presents a statement of a fact, ^ 

^ a concept definition, the- steps of a procedure or ^ 

rule, or a statement of a pijpnciple. 

EXAMPLES : U '*The oharacteri sties pf^a typical jet pump include " 

(concept definition), 

/ y ^ 2. ^^The procedure for changing a gasket/^in a oheck valve is. . • • " 

. ■ . . V— (procedure definition): " , 

> , . • ' / . i/- ■ 

. 3. , '^To deterrrrCne voltage y multiply, current ^y resist^a^ice . " ^ 
■ * (rule statement) . ' > " . 

Component: The student is told or showj; how a statement of a 
^ ' * concept; procedure, rule, or principle applies in a- 

.'I ' , specific case.. 

EXAMPLES : 1. ''The XXYZ pump is a double-acting reciprocating pump ^ 

because it has th^ particular characteigistics noted 

\ ^ on the diagram below. (concept example). ' ^ ' 

2. ''L^t's see how OHM'S LAW applies in a Specific case, ..." ^ ^ 
(rule example)*- 

i ' 3."" ''The Navy's victory at Midway in World' War IT' illustrate s 

' tfie value of cryptologic intelligence be'causei\ . 

- (principle example). - 

PRACTICE REMEMBERING Cpmponent: The student is asked to supply part or all 

of a fapt statement,r concept definition, tiie steps 
of a procedure or rule^ or the statement of a pnVic^ple. 
The student ;,is -given FEEDBACIC about the correctness of 
' ' ' his answer. * r- 

' ■ * ' > •« 

fiJXAMPLES : 1. "The father of our 'country is ^ (Fact) 

2. 'iList in order the steps of procddura for-.t.." (Vrqc^edni^) 

PRACTICE USING Component: The ttudent is asked to use a concept d^f4nition, 

(• procedure, rule, or principle on a specific case. 

> ^ to which it applies, and is given FEEDBACK about / 

- ^ the quality of his performance. • 

EXAMPLES : ' 1., '^Classify the following' Lofargrams." (concept) \^ 

4 ' 

' 2. ''Using the procedureMn the tech\ manual^ dissassemble ^ 
/-'^'-^^ the (procedure) . <v 



V- 



3. "Solve the following circuit ^ problems .... " (rule) 

4. "Predict the effect (sociological^ and psychological) when 

women are hssignejd to Navy ships." (principle) . , 
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For CONSISTENCY, cliffer*ent components are requt»;ecl for different tasky levels: 



l^or* 



th6 REMEMBER level 



For the USE- UNAIDED level 



N 



V 



a STATEMENT . (m le) 

a STATEMENT B^MPLES 

(or a review (at least 

of -the state-^ , one), 
menk^ y 



For the. USE-AIDED level 



aid . , 

place of the 
statement. ) 



EXAMPLES 
WITH AID. 



PRACTICE ' 
REMEMBERING. 

PRACTICE 
USING. 



PRACTICE 
USING WITH 
AID- 



These veqtiired components apply across all content ^types ( facts ^ 
concepts, f procedures J .andl^s, and principles) for REMEMBERING, 
and all except fadts for USING. For example, if^ the objective and 
test item cdHled for the student to remember a fact, then the in^' 
struction musi contain a statement of the fact, to he remembered, 
and at least one prac tice ^remerriberinij item with feedback. No ex^ 
ample is required, because it would be redundant with' the statement. 



\ 



CONSISTENCY also requires that eax;h required component meet the 
following criteria: , ' , 

1. STATEMENTS must be COMPIeTE . . ' ' - 

- « 

2. EXAMPLES must show application of the complete content. 

3. EXAMPLES must match or reflect the conditions and standards required 
of the objective and the test as' closely as possible. 

4. ' PRACTICE must include FEEDBACK . ^ 

5. PRACTICE rmist be of the sameitask/content level as the^test iteoi 
and objective. ^ « 

6. PRACTICE must match or reflect the conditions and standards requiried 
of the objective and the test" as closer^k^as possible, or be designed 
to help the student gradually learn ttte final task. 



MoBt of the requij^ementa above are probably obvious^ but some are 
^complicated. COMPLETENESS, for epmple, requires different pre--^ 
scriptions for different content types: . . , ^ 

. For a CONCEPT : '^complete", weans that all the critical charac- 
teristics of the concept y and their combination, ar^given. 

For a PROCEDMiE : "complete" means that all the steps of the 
^ "procedure aye given in the proper order. "'J 

For a RULE : ' "complete" means that all the steps of the rule/ 
are given in the proper order, 

\ Fpr a' PRINCIPLE: "complete" means that all the pre- and post-- 
conditions, actions, processes, causes, ^effects, and ^results are 
stated, and the relationship betDeen^hem is clearly stated. 



PRESENTATION ADEQUACY 

.Once all the required instructional cofUponents are^^present^ and eaohy^ 
^ of these components meets all of the 'consistency criteria^ the / ^ 
* ADEQUACY of the presentation can be assessed. This is done by^ checking 
each instructional component ( statement ^ examples y practice) for - 
certain chftracieris tics. . ^ ^ 

A STATEMENT is ADEQUATE if it meets the following criteria: ^ , 

The statement must- bp SEPARATED from the rest of the instruction. ' 

This helps the student find the main idea. When^ the statement is 
separatedy^ the key points stand outy and are not buried in the 
presentation. There ^dre several, ways tp accomplish this goal: 



1. 



a. 
b. 
c. 
d. 
e. 



Set-off the statemenjt with boxes. V 
Use a different color. • • e - 

Use \x different type^ or underline. 

Plape on a separate page^ or in a special place on the page.. 
For audio or movies y pause before giving tJie statement. i 

The statement must be ID^IFIED. After the statement is separat\dy 
the student shnuld be told what it is. This permits the studertt^s 
attention to be focused on the key points and th^ir applicationy rather 
than the student trying to become generally familiar with everything in 
the instruction. One way to identify a statement is to. use the word 
"statement." Other more content-oriented words are even more helpful: 



\ 



definition 
Main Idea^ 



procedure for 
Key Point: ' 



the principle of 
General rule: 



EXAMPLE: 



DEFINITION OF OHM'S LAW: 



(Herey the stat^ent 
is separated Jby the 
boXy andjLdentified. ) 



a. 
b. 

c. 
d. 

e. 



3. In. addition to the statement, the presentation should include something 
to help the student better understand and remerriber the statement. 
Metho^ of providing this help include: . \^ 

Giving a MNEMONIC (memory trick). 

Giving a general example of how the statement ^can bemused. 
Explaining why the statement is important. 
Explaining how it came aboupy how it fits in the coursey or 
how it relates to^ something the student already knows: 
Explaining ' some of the terms in the statement. , j ^ 
/. Representing the statement with pictureSy s^mbo^y flowchartSy 
tableSy etc. • ^ 

EXAMPLE: The following figure can be a helpful memory 
device for Ohm's. It will help you remember 
it so^ you can use it later on. 

Other' adequacy covered in the IQI manual ire met.; 




EXAMPLES A^re/ ^DEQUATE if'^^hermeet the following criteria: 



f 



1. EX^itMPLES must be SEf>ARATED and IDENTIPklL 
. -feXAi 



2. 'tXAMPLES must include, aome type of help. 

3. EXAMPlTS sfiould range from "easy" to "hard." ' 



4. Examples .should ^e representative of the job the student will do 
after training. 



5. There should be enough examples to cover the content;?^ea adequately. 

6. I^XAMPLES ^should cl^early show why common errors are wrong. 



The 'criteria are generally selp-^explanatory-, SEPARATED and IDENTIFIED 
are the same aa for statements^ and points 2 to 6 need no further 
ex/tlanation. The second criterion^ HELPj is applied in Aiffereht 
ways for different content types. Some types of HELP for each content 
type are given below: - ■ 

HELPS for CONCEPTS: Highlight the criticaV characteristips of an^ 

example. 

^ Explain why or why^not something i^ classified 

as a member of a concept. 

SHow the use of a checklist or heuristic to 
help classify. 

Simplify early exampleSj e.g.^use line-drawings 
instead of complicated ^photographs. 

HELPS for PROCEDURES or RULES: Explain why each step is done. 

Explain why each step is important. 
- ' ' ' Give additional information about 

how to perform the tasj<. 
Give additional information about how 
' ^ to know if you've done it wrong. 
^ Give flowcharts, tables , .etc. 

HELPS for PRINCIPLES: Highlight important features. 

Simplify the relevant information from the 
case study in which^it is er^edded. 
• . Use logical representations of the IF-THEN 

relationships. 

. Give additional information about how. the 

principle applies^ or why it doesn't, _ 
Give hints as to how to analyze problems. 

Other adequacy concerns are given in the IQI manual. ' 
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PRACTICE itthp are ADEQUATE if th^y meet the following criteria: 
1 1-. . Tfie "practice section mist be SEPARATE and IDEMTIFIEU: 

2. The practice items must be free of hints that.wouldn't be- present . 
in the test or on the job. ' . . 

3. The PRACTICE items should have the same format as the format of 
the test items. 

X.^"' ^ . . ^ • 

4. The PRACTICE items should range from easy to hard. . 
i ^ • - ( 

5. The PRACTICE items should be typical of the job to be performedx 
* after training. , \ 

6^ The PRACTICE items shaytd incl^ude the opportunity for cornmorj errors. 

>. The FEEDBACK must also be SEP^ATED and IDENTIFIED- for each practice 
^ item. ' 

. 8. The^FEEDBACK should include help (simij/rr to that for examples). 

(As a hare minimum, the FEEDBACK slwuld -dir^at the student baak to 
where the instruction was originally' presented. However, it is 
better to have a new brief presentation, because if the studeni 
got the practice wrong, the original presentation didn't help enough.) 

The criteria are also self-explanatory. 

Other adequacy concerns are given- in the IQI manual. 
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the G(vne sub j eat matter, ) 



\ 



INADEQUATE PRESENTATION on .the pHncirlea of operation of an alarm circuit 



%l^lee^ I 



- J : - 

The alarm circuit senses extremely high temperatures-. When an 
extreme steam temperature occurs (which is a very dangerous condihion 
'that may, have, a/flverse consequences for « ship and her crew), the sensing ' 
switch contact^ close .thus shunting the resistor. The decreased resis- ' 
tance in the di'r/uit, according to OHM'S LAW (E=IR), causes an increase 
in current flow/in the circuit, which is enough to operate the alarm 
relay. The re^ay is designed to~operate at a current flow above that 
normally found in the circuit. OHM'S LAW states that with yoltage 
constant, a decrease in resistance~tn the circuit must be accompanied 
by an increase in current flow.\ The contacts of thie alarm relay then 
closje to actuate the audible alarm-slevice, which may consist; of a warhihg 
bell with an electrically opera-ted clapper, or an H254 resonated horn 
assembly. Both of these produce extremely loud signals so they can 
overcome normal ambient noise levels. 



A" 



J L 



to external audible alarm device, 



Why is it important that the alarm circuit be operational at all times? 
Remember what hot steam can do to ships and sailors. 



The example above is inadequate in' several ways. Firsts the principle 
of operation of the airouit is not separated or identified. How is the 
student to know what to learn from this presentation? Second^, the 
presentation is cluttered with a lot of other ''nice to know" information 
that really doesn't help. If helps were included, they should aid 
remembering or understanding the principles of operation of the circuit. 
Also, the practice is not separate or identified, there is no feedback, 
and the practice really has nothing to do with remembering the principle. 



The next page shows a more adequate presentation. 
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MORE ADEQUATE PRESENTATION on the pHnJcplea of operation of an alarm circuit: 



OPERATION OF THE ALARM CIRCUIT: 



(This 'section deecribee how 
tfie. alarm circuit operates) 



BASIC SCHEMATIC 



Extremely high steam tenjiwatures cause the 
switch to close. This shujts the resistor, 
bfecause the switch and the resistor are 
connected in parallel. Circuit resistance 
is decreased" and, therefore, current flow 
is increased. .The increased current flow 
operates the relay, closes its contacts, 
jind energizes the bell or horn. 

EXPLANATION 




PRACTICE: 



1. High temperature closes switch. 

2. Switch shunts resistor. 

3. Decreased resistance = increased 
current flow. (OHM'S LAW) 



rylliy. 



4. Increased current operates re 

5. Relay contacts close. 

6. Relay contacts energize bell or Jiorn, 



Without using, re fevenoea or notes y explain hou an alarm circuit 
operates. Be sure to include in your explanation the important 
actions that take place in the circuit. (Answer on pg. 256. ) 



256 

ANSWER TO PRACTICE QUESTION: There are several ways you could have explained 
the operation of the alarm circuity but your answer should have included 
the following ideas: 

1. High temperature causes the switch to close. 

2. When th^ switch closes it reduces total resistance 
in the circuit. 

J. Decreased resistance means increased current flow. 
4.^ The increased current, flow operates the relay, 
V J 5-^ The rdlay contacts close and operate the hell or 

horn. 

\ 





USING TIE IQI: . * ' 

? ^ ■ 

l\\e IQI is *signed for Qual ity Control during any object ives-based 
• instructional developmervT process. 

There arefoyr documents that comprise the IQl) 

1. Introduction and Overview /Tkie doaument) ' 

2. User's Manudl (contains all IQI procedures, and examples 

^ of their use) • ^ • 

3. Workbook (contains practice on all IQI procedures, with 

feedback)_ 

4. Job f^rformance 'Aid (short version of all procedures) 

To facll/tflitfe use of the IQI procedures, the User's Manual, Workbook, 
^ and JPA Were designed to include three quality control forms: The 
first form assesses objective adequacy, and the second is used to' 
determine test consistency and test adequacy, and the third evaluates^ 
objectiveVpresentatiOn consistency and presentation adequacy. The * 
suggested of these forms fs as follows: 

1. EitherNkiring, or inmediately after, the development of 
objectivo^^in instructional development, use the objective 
adequacy form to assess the adequacy of each objective'. 
Any required revisions should be made before instructional 
development proceeds. 

2. As test items are developed for each objective, they should be 
checked for consistency with objectives, and adequacy, using 
the second form. 

3. As new Instructional materials are developed, or as existing 
^ * materials are adopted, they are checked for consistency. wittr 

objectives, and adequacy, using the third form. Required 
revisions to materials and tests are made before they are 
^ subjected to individual or small -gfoup try-outs. 

FOR M3HE irWRmTIOfi: 

'X 

For more informatiofi on the IQI, contact: phone: 

riavy Personnel Research and Development Center (/M) 225-7121 

Code P304 * - 

San Diego, CA, 92152 AUTOVOfI 933-7121 

7140 
7194 
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